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” Disclaimer 


Presentation Disclaimer: All opinions, judgments, recommendations, 
etc. that are presented herein are the opinions of the presenter of the 
material and do not necessarily reflect the opinions of the PCI-SIG®. 
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PCle and Optical Interfaces 


= Today there's no complete specification for PCle over optical 
interfaces 


v Current optical interfaces are not optimised for PCle use 
v Often designed for proprietary interfaces or other non-PCle standards 
v This leads to issues with 
— Supported frequency ranges 
— Serdes levels and impedances 
Squelching of quiet periods 
Lane delays 
— Latency 
— Side band signals 
v Future cable specifications such as OCuLink and PCle 3.0 Cable 
Spec include the possibility of using optical interfaces within 
them 


— As these are not yet released currently both specs are short on detail with 
regards to how AOC's will interface with PCle devices 


— Methods listed here can potentially be used with these new standards 
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~~ PCle and Optical Interfaces 


= Most testing has been done with Avago/PLX 
devices as link partners 


v Limited number of other PCle devices have been 
shown to link 


v All optical links have required some “tuning” of the 
serdes to provide a reliable interface 


v Testing and evaluation is required on the optical 
modules used before rolling out a design 


v Reliable link up can be achieved but it requires some 
work 
= Aim of this presentation is to provide you with 
hints and tips on how to achieve reliable links 
through optics 
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Optical Interface Options 


= Many types of Optical interfaces can be used 
v We’ve tested many of these by many manufacturers 


= Active Optic Cables tested include 
Ÿ SFP 
v QSFP+ 
v Mini-SAS HD 
v |-Pass 
v Etc. 

= Direct attached include 
v MiniPod 


v MicroPod ETS | AA 


v Etc. 
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~~ Challenges of Using Optics 


= Optical interface must be able to handle the 
frequency range of PCle 
v Everything starts at 2.5 GT/s then switches to the 
appropriate speed 
v Optical interfaces have to handle not just the final link 
speed required but also the 2.5 GT/s link speed 


— Some optical interfaces require selection of the appropriate 
bandwidth range via straps 


— Important to ensure that 2.5 GT/s link speed is included 
within the same bandwidth range as the final target link 
speed 

v This is not normally an issue for most modules as 
they don't do re-timing 
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= Receiver Detect 
v |Impedances of Optical interfaces often confuse RD 
circuitry 
— Generally terminated at 50 / 100 ohms, even 
nan unpowered in many cases! 
— Termination scheme can inhibit detection as may be 
presented in a non-compliant manner (cf PCle spec.) 
For example many are not terminated to Ground 


v Ifthe Receiver Detect circuitry doesn’t see the 


RD detection circuitry in TX detects appropriate impedance 
change in pulse shape caused by AC $ . ae Ts Z 
ne and termination Will not begin link training — stays in Detect state 


— May link sometimes but not others 


— Some lanes may link and some may not reducing link 
width 


v Needs to be tested on the serdes/optics used in the 
system 


— lf possible have the ability to mask receiver detect in 
the PCle link partners 
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~ Quiet and Idle Time 


= Quiet or idle periods on the link where there is no traffic 
being sent can cause issues 


v This can occur when: 
— link is between detect and sending training sequences or 
— if the link goes into electrical idle caused by power management or 
— link is retraining due to speed or link width changes 
v Typical module specifies startup in milliseconds... 
v Result of this is: 
— Absence of optical modulation can saturate the PIN diode (RX) 
— Chatter can occur (No squelch) 
— Link can take significantly more time to stabilize due to 
e False locking 
e False symbol detection 
— Can result in false detection of Training Compliance Bits 
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Optical Saturation 


= Example of ‘dead-time’ saturation in optics 
v Test example of Avago Minipod optics 


v Test consisted of a pattern generator feeding an 
Avago T X->Laser->PD->AvagoRX->Scope 


v Run patterns of varying non-transition and transition 
lengths to simulate varying times of EIDLE feeding 
the optics. Observe recovery time. 


v Test included multiple lanes and TX/RX on each 
Minipod 
v Several patterns tested but only one example shown 
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Ba Optical Saturation 


= Pattern is 100K clocks, 200K zero’s, 100K clocks 


Settli 
Zong 


SBR 
HT 


L 


ac. Additional Saturation SE 
Comments 
= Settling time can vary markedly between lanes 


on an optical link and between optical interfaces 
of the same model 


= Early link noise can confuse PCle serdes 


v Can delay link up time considerably 
— Maybe by minutes 


PCI-SIG Developers Conference Copyright © 2015, PCI-SIG, All Rights Reserved 12 


PCI ° 


a, ns 


PCI 
EXPRESS 


~ Quiet and Idle Time cont’d 


Disable ASPM to prevent low power link states 
v Usually possible to set in device registers or in the BIOS/OS 
Disable autonomous link/width speed changes if possible 
v Usually possible to set in device registers 
Reducing the Detect.Quiet Wait Time if possible 
v Requires access to Phy settings 
v May not be needed depending on the Phy 
Disable TCB bits once compliance testing completed 
v Requires access to Phy settings 
Take care using Squelch or other optical serdes features 
v Well designed squelch and AGC should improve link up 
v Badly designed AGC may cause issues and add delays 


— Delays can cause missed training sequences or malformed 
TLP’s 


v Keep link active if possible to avoid need for these 
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~ 8 GT/s Challenges 


= Optical devices are non-linear 
v TX Side 


— TX backchannel tuning may not work or may give unreliable 
results 


— May need to bypass training phases 

— May need to use fixed presets or fixed coefficients 
v RX Side 

— Optical module and trace length dependent 


— It is likely that CTLE values of ATT & Boost will need to be 
either: 


e Modified to have a specific start value or 
e Locked to a fixed value 
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Tuning Performance 


= Optical interface may require lower input levels to 
prevent receiver saturation 


v May need to reduce the overall transmitter amplitude levels 
= If possible capture the eye on both sides of the link 
v Ideally use serdes built in eye capture after the equalisers 
= Tune transmitter and receiver parameters to obtain best 
possible eye 


v Inthe case of 8 GT/s lock down transmitter and receiver 
parameters and prevent auto-tuning 


v May be able to get away with just setting start values of CTLE 
and allowing CTLE tuning in some cases. 
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~~ Tuning Continued 


= Bigger isnt necessarily better 


v Excessive amplitude in the receiver can cause 
saturation of the serdes slicer 


v Attenuate in the CTLE to ensure this doesn’t happen 


v Eyes and their limits will vary between serdes 
vendors so contact serdes vendor for advice 


v Eye extraction is sub-sampling so check with error 
counters etc to determine link quality 


ATT 8 / Boost 3 ATT7/ Boost 3 ATT5 U Boost 3 ATT 4 I Boost 3 ATT 3 / Boost 3 
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~ Additional Changes 


= In addition to disabling receiver detect, if possible use 
inferred electrical idle detection 
v Rely on data rather analog threshold to change speed 
v Forces TX training sets 
v Will require access to Phy registers 
= Down-train Disable — inhibits width reduction 
v Will require access to Phy registers 
— Only needed on one end of the link 
v Optical lane ‘start up’ time varies 
v Downtrain disable forces retrain if needed 
— Link training may fail on some lanes if that optical lane isn't ready 
— Retrain fixes this issue 


PCI-SIG Developers Conference Copyright © 2015, PCI-SIG, All Rights Reserved 17 


a Poe. 
Sideband Signals 


= Optical interfaces may not have capability or 
Spare capacity for: 
v PERST# 
v REFCLK 
v Power On 
v WAKE# 
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= Sideband Signals 


= Not obvious how you can push PERST# and 
other low frequency sideband signals down 
optics 
v Possible options are 


— Use Autonomous resets e.g. Power good on downstream 
devices/cards 
e Not ideal as cannot generate a PERST# to the remote device 


— Use a Spare optical link(s) or run a sideband cable(s) along 
with the high speed optics 


e Increases cost of the interface 
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~ Sideband Signals cont’d D 


= Many optics have I2C/SMBus interfaces or sideband 
signals which can be used to assist 
v Use light presence and/or modulation present to generate 
PERST# 


— E.g. use GPIO or other signals to control optical enable at one end 
and use “light out” indication at the other end (perhaps AND’ing 
signals from lanes) to generate the PERST# or other signals 


v Use 12C/SMBus and special LED patterns to generate PERST# 
— May require FPGA or CPLD to provide additional logic 


— Can generate PRSNT# and other low frequency sideband signals 
in the same manner 


v These methods have been used/demonstrated by Avago/PLX 
and several customers 
— No standard yet but emerging standards do have access to the 


optics via 12C/SMBus e.g. Cable Management Interface in Gen 3 
cable spec. 


— Currently locks user to one type of optics 
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=" Sideband Signals - REFCLK 


= Same issue as for the other sidebands 

v i.e. may need another cable/optical link for the clock 
= Asynchronous REFCLK operation possible 

v Pre-SRIS compliant 


— Need a low noise non-SSC REFCLK on each end of the link 


— SRNS compliant REFCLK should work but would need testing on 
pre SRIS devices 


— If you can't guarantee the noise or the REFCLK type use SSC 
isolation capable devices and run the REFCLK seperately 


v SRIS Compliant 
— Needs SRIS compliant REFCLK at both ends 
— Needs SRIS compliant devices at both ends 
v As long as REFCLK noise is low, link up is normally possible 
— Avago/PLX Switch to switch is very reliable 
— Asynchronous operation is the most commonly used method 
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Reliability 


= Once tuned optical links have proven very 
reliable 

= High reliability designs should ideally have: 
v an (optical) PERST# mechanism 
v and/or watchdog for link status 

= Monitor Error registers to validate optimal 


settings and to check for changes in 
performance 


v Error counters are very useful for this and can be 
used to set alarms or bring the link down in the case 
of excessive errors 
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” How Far Can You Go 


= Longer optical cables add delay 
v Adds latency 
v Eventually affects ACK/NAK and updateFC protocol 


— Reduces performance 
— This affect also seen when adding re-timers or other passive 
delay elements 
= Tested up to 50m of cable 
v 100m should be possible without issues 
v Few km (1-2) probably Ack/Nak limit but haven’t 
tested 

— May be able to get further by bending the spec. 


— Likely to be throughput hits on an x4 link as you get over a 
200m 
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~~ Can You Use Any PCle Device 


= Most testing has been with Avago/PLX switches 
v All types over all generations 
v Serdes are highly configurable 
= Off the shelf end points are unlikely to work without 
assistance from the vendor 
v Not seen any work yet 
= Re-timers have been seen to work in one case but 
required lots of assistance from the vendor 


= FPGA’s have been seen to work but customer had to 
adapt the Phy settings as per the previous notes 


= Custom solutions have been created to allow PCle 
interconnect but require additional ASIC’s or adapter 
cards 
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Customer Issues 


= The link was working fine without masking receiver 
detection. Now it doesn't. Why do | now need to mask 
RD? 
v For some systems, we see TS1s driven from the PCle device to 


the adapter and then after 24ms or so, it times out and sends a 
compliance pattern 


v When the adapter is replaced by another card or optical cable 
we go immediately into training without any compliance pattern 
sent 


= Conclusion: Optical Modules are not intended for PCle 
and there can be variability between modules (even 
within same model/manufacturer) 
v Masking RD allows for consistent link training across a range of 
devices/modules 


PCI-SIG Developers Conference Copyright © 2015, PCI-SIG, All Rights Reserved 25 


eac. S 


Customer Example 


= Using dual x4 AOC devices to create an x8 link 


= Initial issue was inability to attain an x8 link 

v Achieved operation without bit error 

v Minimized link training time 

v Steps to achieve this were as per previous foils: 
— Basic masks — RX detection / EIDLE (Attain Link) 
— Down-train inhibit ( Force max width) 
— TX/RX calibration ( Link bit error reduction) 
— Detect Quiet delay reduction (improve multi-lane linkup time) 
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Active Optical Cables 


= One x4 AOC okay 
= x8 using two AOCS fails to link correctly 


v Dual x4 Optical modules used to create an x8 Optical 
link 
v Independent Reset / Power testing 


v Max x8 Link time seen to be seconds to minutes for 
all 8 lanes! 


10m 
PLX 
x8 Switch 


PCI-SIG Developers Conference Copyright © 2015, PCI-SIG, All Rights Reserved 


PCI ¢ PCI 
EXPRESS 


T 
p 


Steps To Get Link Up 


= First test the optical modules to ensure these operate 
correctly 


v Test each AOC individually to 
= As per previous foils 

— Switch to inferred Electrical Idle 
— Mask Receiver Detect 
— Disable down train 

v Link now comes up as x8 but not 100% of the time 
— Tune eye s using serdes C TLE changes 
— Tune AOC output (doesn't normally help much) 

v Receiver errors gone but still not linking 100% of the time 
— Reduce Detect.Quiet wait time 


= Successful link 100% of the time across all devices 
tested 
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Detect Quiet 


= Reducing the detect.quiet time reduces optical 
module saturation 


v See Foil 12 
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~~ Which Optical Devices to Use 


= We've tested many manufacturers optical 
devices including 
v Avago 
v Finisar 
v FCI 
v Samtec 
= All have worked with appropriate serdes 
changes 
v Some only required minimal changes 
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ac. Flexible Application ner 
Optimized Server uhai 


PCle 3.0 connection 
via Silicon Photonic 
MXC cables. a 


Composed server 


= Application Optimized 

v GPGPUSs appear direct attached 

v IB controller appear direct attached 
{v  PCle SFF SSD appear direct attached 
4 


Storage HBA and HDD’s appear direct 
attached 


Peer to Peer communication e.g. GPGPU to 
GPGPU to IB controller to PCIe SFF SSD 
possible IR HII 
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Examples of Test Boards 


Test boards using Micropods 
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___ Tested Optical Interfaces 


Optical/Connector | Part Number 
Type 


Avago QSFP+ AFBR-7QERxxZ 

Avago MiniPOD AFBR-81 xxxZ/82xxxZ 

Avago MicroPOD AFBR-77DxxZ 
/78DxxZ 

Finisar BOA FBOPD10SL1L01 

Finisar QSFP+ FTL410QxxC 


squadwire 
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” Tested __Tested Optical Interfaces 


Optical/Connector | Part Number 
Type 


Finisar C.Wire FCBGD10CDxCxx 


eS s 


FCI MiniSAS HD 10117949-xxxxLF Ne 


101231 96-xxxxxxx D 
Le a 
Samtec* PCle iPass PCIEO-wGs-xxxx.x K DSN 


7 D PCIEO-4G2--005.0-+ = 
X 
ne S 


= Many other optical interfaces tested and shown 
to work by both Avago and various customers 


*Some issues with 2.0 devices but this may have been an early firmware revision from Samtec. Customers have shown 3.0 to work with these 
cables. 
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Summary 


= PCle over optical cables works and has been 
used in production for many years now 

= Currently serdes changes are required for the 
majority of optical cables 


v This requires some programmability in both PCle link 
partners serdes 


v Testing and validation required for systems 
v Currently locks systems to known devices and optical 
interfaces 
= New specs such as SRIS, OCuLink and the 3.0 
Cable Spec should provide increased 
opportunities for linking over optical cables 
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Thank you for attending the 
PCI-SIG Developers Conference 


Israel 2015 


For more information please go to 
WWW.pcisig.com 
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Appendix 


= Useful Avago/PLX switch registers 
= Receiver Detect and Electrical Idle Detect — 204h 
= Downtrain disable 
¥ BE4h (station-based Phy Port Chicken Bits 0) 


— [5:0] (Downtrain disable — one bit per 
port of the station) 


= Detect Quiet 


v  220h (station-based Physical Layer Function 
Control) 


— [9:8]=0 (reduce the Detect.Quiet Wait 
Time from 4 mS (default) to 0 mS) 


= TCB disable 

y 224h (station-based PHY Layer Test 0) 
— [9] Ignore Compliance Receive TCB 

v  22Ch (station-based PHY Layer Chicken Bits) 
— [20] Ignore Hot Reset TCB 
— [21] Ignore Disable Link TCB 
— [22] Ignore Loopback TCB 
— [23] Ignore Disable Scrambling TCB 
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= Registers to monitor for errors 
v Link Status 


— 0x160[17] (VCO Negotiation Pending) — 
most accurate method to determine link 
status 


v RX Detected Errors 
— 0x724 (Framing Errors) 
— 0xBC4[15:0] (Recovery Counter) 
— OQxBFO (Receiver Errors) 
— OQxFAC (Bad TLP Counter) 
— OQxFBO (Bad DLLP Counter) 
v TX Detected Errors 
— OQxFC4[12] (Replay Timer Timeout) 
— OxFC4[8] (Replay Num Rollover) 
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